Sampling traces without lying to yourself

Tracing every request is affordable right up until it is not, and the moment it stops being affordable is usually the week you most need the traces.

So everyone samples. The question is which requests you are willing to never see again.

Head sampling, and what it costs you

Head sampling decides at the start of the request, before anything has happened. It is one random number and it is free, which is why it is the default everywhere.

It is also blind. A 20% head sample of a route that errors once in five hundred requests keeps, on average, one in every 2,500 failures — and the slowest request of the day is exactly as likely to be dropped as the fastest.

What we do instead

That is tail sampling with a short buffer: we hold the spans for the length of the request, then decide. It costs us memory on the ingest side, which is our problem, and it costs you about 38 ms at p95 in the SDK, which is the honest number and the one we publish.

Reading a sampled percentile

Two rules that will save an argument in a review.

• Rates are fine, counts are not. A sampled p95 is a good estimate. A sampled count of anything is a guess multiplied by five, and it should be labelled as one in whatever dashboard you paste it into.
• Never compare across a rate change. If you moved from 100% to 20% on Tuesday, Monday's graph and Wednesday's graph are two different measurements wearing the same axis.

Artisan marks every chart drawn from sampled data with its rate, and refuses to draw a count from anything under 100%. We would rather show a gap than a confident wrong number.